Restart files for an ne120 run: to facilitating making high-res transient compsets
==================================================================================
Philip Cameron-Smith (updated 2019-11-6)

Provenance of ne120 restart files
---------------------------------

[From an email from Peter Caldwell to Philip Cameron-Smith, 2019-5-7]

Files needed for restart should be under:

/global/cscratch1/sd/petercal/E3SM_simulations/theta.20180906.branch_noCNT.A_WCYCL1950S_CMIP6_HR.ne120_oRRS18v3_ICG/run/

restart time is 0056-01

I haven’t copied all history files for the last timestep, but I don’t think that matters (?).

The run_e3sm script is at: https://github.com/E3SM-Project/SimulationScripts/blob/master/archive/A_WCYCL1950/ne120/run_e3sm.1950.theta-v1release-branch-noCNT.csh 

I think switching theta to cori should be sufficient for getting going (i.e. I think there is a default PE layout). If not… I guess we need to ask Noel. 

Provenance of ne120 initial file
--------------------------------

A hybrid restart is needed in order to start from a date different to
that in the restart files.  This means the atmosphere needs to use an
initial file rather than a restart file.  The initial file was
generated by running the 1950 compset for a day and setting
inithist='DAILY'.  The file generated was:

  1950_t1.A_WCYCL1950S_CMIP6_HR.ne120_oRRS18v3_ICG.maint-1.0.cam.i.0056-01-02-00000.nc

SEAICE XTIME ERROR
------------------

The mpascice restart file contains a variable called 'xtime', which
contains the model time for the data in the file.  The seaice model
reads the xtime variable and complains that the xtime doesn't match
the current model time.  The mismatch produces an error message in the
log.seaice.nnnn.err files.  The simple solution is to rename the xtime
variable in the original netcdf file to xtime.orig:

  ncrename -v xtime,xtime.orig mpascice.rst.0056-01-01_00000.nc

The error produced without the fix is shown below:

>> more log.ocean.0003.err
----------------------------------------------------------------------
Beginning MPAS-ocean Error Log File for task       3 of     960
    Opened at 2019/11/06 19:24:08
----------------------------------------------------------------------

ERROR: File mpaso.rst.0056-01-01_00000.nc does not contain the time 1950-01-01_00:00:00