I don't think the scratch system is working as advertised to MsPASS course students

GeoLab Help & Troubleshooting
1

I was doing some followup on the MsPASS course taught using geolab last week. The moodle documents the students had to read claim GeoLab has a simulated large scratch file system available at the mount point “/earthscope-scratch”. At least when I load the MsPASS image I do not see that file system with any shell tool I know of. What does work is this shell construct:

aws s3 ls $SCRATCH_BUCKET

That yields an obscure UID name:

 PRE auth0|66620230da0803120dd94aaf/

I can read and write data to the scratch bucket with this kind of rather complicated python code:

import os
import s3fs
import pickle

scratch_uri = os.environ.get('SCRATCH_BUCKET')
s3 = s3fs.S3FileSystem()
s3_path = f"{scratch_uri}/test.dat"
with s3.open(s3_path,"wb") as f:
    # d is a TimeSeries object read earlier - content is irrelevant for this post
    pickle.dump(d,f)
# read it back
with s3.open(s3_path,"rb") as f:
    d2 = pickle.load(f)

which run successfully and I can confirm d and d2 are identical. In short, there is a scratch system but it is not (maybe yet) implemented as a virtual file system.

Am I doing something wrong or was the claim that /earthscope-scratch could be used vaporware?

2

Hi @pavlis ,

It sounds like something may have been miscommunicated. For now, /earthscope_scratch is not a virtual filesystem, it’s an S3 bucket adjacent to GeoLab. It may be possible to mount it as a filesystem in the future, but we have not yet taken those steps.

What you exposed with the aws s3 ls command in your post is the prefix for the partition of the bucket that’s allocated to your unique user ID.
If you try $ echo $SCRATCH_BUCKET you should get s3://earthscope-scratch/<Your_user_ID>

You can use the approach you’ve shown above in your post, or other tools like boto3 to access the bucket using your prefix.

3

Thanks, Sarah. Someone put in the moodle site for the mapass course that here was a /earthscope_scratch mount point. As ou say that apparently was misinformation. As you say there are ways to work with this with boto3 or s3fs. Got lots of good suggestions from Gemini. Does present a problem for mpsass but only because we had no motivation to write data to s3 until now.