Research Computing
Frequently Asked Questions (FAQs)

Haas management has decided that Research computing needs to be managed by two different departments, One department manages windows based research computers and the other department manages Linux based computers. Please click here to view the services offered to reserachers and instructors.

In general, anytime you have a question or an issue to report, you can email helpdesk@haas.berkeley.edu. If you wish, you can contact tony_cricelli@berkeley.edu directly since he is the web developer and database administrator for this site.

Haas maintains four general use clusters:

• Ubuntu-based Linux Cluster
• Microsoft-based Windows Cluster
• Kubernetes Jupyterhub-based Cluster
• Analytic Environment on Demand (AEoD) for custom Virtual Machines

Haas maintains the following software packages
• Matlab
• SAS
• Python
• R
• C,C++,Fortran
• Julia

In general, Haas has systems administrators, not programmers. The Haas administrators main duties are to keep the computer systems updated and available 24/7. Systems administrators also ensure that research software applications are fully-functioning. It does not hurt to ask, the Haas administrators have helped thousands of students debug codes over the years. Send an email to helpdesk@haas.berkeley.edu and ask for a Haas Research Computing staff member for assistance.

The Berkeley Data Lab (D-Lab) provides consulting for research applications and programming languages. Go here D-Lab Consulting to schedule a consulting appointment.

Jupyterhub is a web interface to other software such has Python, R, SAS, MATLAB, etc.... R and Python are the main two languages used.

Kubernetes is the software we use to manage our Jupyterhub teaching workload. It manages containerized workloads and services automatically.

Yes, we use Kopia to do encrypted snapshots every 12 hours. The snapshots are stord in the Google Cloud.

Yes, we use Kopia to do encrypted snapshots every 12 hours. The snapshots are stord in the Google Cloud. We will keep the last 60 snapshots which allow us to go back 30 days.

Yes, we use Kopia to do encrypted snapshots every 12 hours. The snapshots are stord in the Google Cloud. We will keep the last 60 snapshots which allow us to go back 30 days.

Yes, we snapshot research data located on our HPC cluster to Google Cloud every 12 hours. In case of emergency, we can spin up a cluster on Google cloud and restore latest snapshot. Our specific procedure is: we would allocate a Google Filestore large enough to be able to restore data. We would then allocate a Linux based (Ubuntu 20.+) node and mount our Kopia snapshot to it. We would first mount the /etc snapshot to recover all accounts and passwords. Then we would mount the /apps folder to recover all installed apps and finally we would mount the /home snapshot and have immediate account to all research data. Then we would mount the Google Filestore and start copying the research files from the mounted /home snapshot. We estimate this would take 4 hours to complete.

Yes, below are the basics to get you operational.

1. Getting Started with the Terminal

The terminal is where you type commands. You can open it by pressing Ctrl + Alt + T or searching for "Terminal".

user@computer:~$

This prompt means:

• user – your username
• computer – your machine’s name
• ~ – your home directory

2. Navigating the File System

pwd        # Show current directory
ls         # List files
ls -l      # Detailed list
cd /path   # Change directory
cd ..      # Go up one directory
        

3. Managing Files and Directories

touch file.txt       # Create a file
mkdir myfolder       # Create a directory
cp file1 file2       # Copy files
mv old new           # Move or rename files
rm file.txt          # Delete a file
rm -r folder         # Delete a folder
        

4. Viewing and Editing Files

cat file.txt         # Display contents
less file.txt        # Scroll through contents
nano file.txt        # Edit file (Ctrl + O = save, Ctrl + X = exit)
        

5. File Permissions

ls -l                # Show permissions
chmod 755 script.sh  # Change permissions
        

6. File Transfers with sftp

sftp username@hpc.haastech.org

# Inside SFTP:
ls            # List remote files
pwd           # Remote directory
lcd /path     # Local directory
get file.txt  # Download file
put file.txt  # Upload file
exit
        

7. File Transfers with scp

# Upload a file
scp localfile.txt username@hpc.haastech.org:/remote/path/

# Download a file
scp username@hpc.haastech.org:/remote/file.txt /local/path/

# Copy a directory
scp -r myfolder username@hpc.haastech.org:/remote/path/
        

8. Other Useful Commands

man ls      # Show manual for ls
history     # Show previous commands
clear       # Clear screen
exit        # Exit session
        

Please email helpdesk@haas.berkeley.edu and request access to the Haas Linux research cluster.

In order to access the Haas research cluster, you must be on the U.C. Berkeley network or have the Berkeley VPN connected. SSH and SFTP are the only two connection methods supported.

There are many ssh and sftp clients available so it is mostly personal preference.

• MacOS ships with ssh. Many researchers, open a terminal windows and enter: ssh haas-hpc00.haas.berkeley.edu
• Windows has two main clients researchers tend to use, putty and mobaxterm, both freely downloadable. Download puTTY here: puTTY Downloads. Download MobaXTerm here: MobaXTerm Download
• Fast-X is an excellent SSH client that accelerates GUI-based applications such as XSTATA, RStudio, Spyder and others. FastX is available for Windows, MacOS, and Linux. Download here: FastX Downloads

Yes open a ticket by emailing helpdesk@haas.berkeley.edu You can also contact tony@haas.berkeley.edu or zane@berkeley.edu directly. We will do our best to install any package that is available to us and does not break the system.

Yes ! You may create your own Python environment with this command: conda init bash

Yes ! There are several ways to run it. Below are the setup directions: Two Steps in preparation to launch jupyter that only have to be done once:

Or just go to "https://jupyter.haastech.org" and skip all the steps
or
just enter "source activate jupyter" and go to step 2

1. Create your virtual Environment: conda init bash conda create -n my_new_environment_name source activate my_new_environment_name 2. Pick a browser that you normally do not use on your computer. Some browser names are Chrome, Brave, Disenter, FireFox, Opera, Edge, etc.... This browser is going to be set up differently than your normal browsers. It is going to be set up so that it uses a "proxy". The proxy is going to be created by ssh. I chose firefox as my jupyterhub browser. After starting the browser: a. Click on the 3 horizontal lines on the upper left of the browser (often called the "hamburger") b. Click on preferences c. Click on General on the right d. Scroll to the bottom of the page e. Click on settings under network settings f. Select Manual proxy configuration g. Click on Socks Host box and enter 127.0.0.1 h. In the port box put a number in, like 3456 i. Click OK and save it. What we just did is force the browser to "surf" through localhost port 3456.

Next are the steps to launch a jupyterhub-notebook job to the cluster and use ssh to connect your local browser to the HPC computer node running jupyterhub. 1. ssh to hpc.haastech.org: ssh -C -D 3456 username@hpc.haastech.org 2. launch jupyterhub: /apps/bin/jupyterhub (make note of JobID) 3. run bpeek command after about 10 seconds waiting for jupyterhub to start: bpeek jobid Towards the bottom of the output, you will see something like this: http://haas-hpc10:2225/?token=72abceasyas1230b3b52a2220055eb1662f628e12707c5e3 4. The final step is to enter the above URL into your home browser that is being proxied. But you will run into a problem because your home computer does not know what haas-hpc10 is. From home you have to put the IP address, so haas-hpc10 becomes 10.10.10.20. Examples: haas-hpc01 becomes 10.10.10.11 haas-hpc07 becomes 10.10.10.17 haas-hpc08 becomes 10.10.10.18 haas-hpc09 becomes 10.10.10.19 haas-hpc10 becomes 10.10.10.20 So the output of the third step was: http://haas-hpc10:2225/?token=72abceasyas1230b3b52a2220055eb1662f628e12707c5e3 but you change it to this: http://10.10.10.20:2225/?token=72abceasyas1230b3b52a2220055eb1662f628e12707c5e3 5. Enjoy your custom jupyterhub! Since you are in a virtual environment, you may add custom languages and custom libraries. You can close your browser at any time and return days later and pick up where you left off.

The cluster offers several queues for users of varying sizes. The command to use is

bqueues

. To submit a job to a specific queue use the bsub command with the -q option. For examples:

bsub -q Queue Name

.

Sometimes this question comes up and is tricky to answer. It is akin to asking, I have access to a Boeing 747, how do I fly it? The answer could be, first sit in the pilots seat ... The cluster is a computer, you can edit files, store files, write programs in many languages, run interactive jobs, run batch jobs, etc...

The Linux cluster has the following softare installed:

STATA, SAS, Matlab

R, Python, Julia, C, C++, Fortran

Bash, Perl, Awk

Emacs, Vi, RStudio, Spyder

Jupyter Notebook, JupyterHub

Yes, put the following in a file, but change the cd command, the error output file name and the console output file name.

			#!/bin/sh
			#BSUB -q special                    # Queue to submit job to
			#BSUB -n 1                         # Ask for 1 core
			#BSUB -e product_server_err.txt    # error output from script
			#BSUB -o product_server_out.txt    # console output from script
			#BSUB -Is


			#
			# Change your working folder
			#
			cd /home/tony
			
			#
			# run system R with my program
			#
			/apps/src/R-4.1.2/bin/R   tony.R
                   

Then to submit the job enter this command:

		    bsub < scriptname
		   

I try to use discriptive names, like R_program.bsub, but it is totally up to you.

The FastX clients are here:
StarNet Download Site
Once you have fastx installed, on startup you should see something like this:

Click the + and add your information:


Click OK then double click on the line that shows up:



On the next screen you will see another + sign:



You will see:



I use XFCE as my GUI.
That will get you this:

All the setup will be saved, next time you start up fastx, it will pick-up where you left off. All the above setup is done just once.
You may safely ignore this window if it pops up. We do not run the webserver part of FastX

MS Windows SSH/X11 Software (Alternative to FastX)
MobaXterm Home Edition Download Site
This program will allow you to ssh into the Haas HPC and run graphical programs from the HPC.
Easy to use, download, install and run it.
Click on ssh create a session. The DISPLAY variable will be automatically set.

Send an email to helpdesk@haas.berkeley.edu with your CalNet ID.

Use a remote desktop client to connect to TS-Research.haas.berkeley with your Calnet ID and your passcode.

Send an email to helpdesk@haas.berkeley.edu with your CalNet ID.

Send an email to helpdesk@haas.berkeley.edu with your CalNet ID.

Send an email to helpdesk@haas.berkeley.edu with your CalNet ID.

Send an email to helpdesk@haas.berkeley.edu with your CalNet ID

Send an email to helpdesk@haas.berkeley.edu with your CalNet ID.

Yes, you can save your files or folders on your documents and download folders. Make sure to back up your files or folders.

Click on the "LogOff" button of the desktop

ANACONDA Navigator
IBM SPSS AMOS
IBM SPSS Statistics
MATHEMATICA
Matlab
Python
R
Rscript
SAS
STATA SE/MP

The AEoD cluster itself is not on the public Berkeley network so it is not accessible. The VMs we install for you are accessible via https://citrix.berkeley.edu -- sign in with your CalNet ID and passphrase. Then look for your VM under the "Desktops" section.

Please send an email to helpdesk@haas.berkeley.edu or zane@berkeley.edu with your requirments: Operating System, RAM, disksize, backup requirements, software to be installed, admin access, Calnet IDs of users who need access, etc...