<!-- #region -->
# Organizing your work

-----------------------------------
> Your closest collaborator is you, six months ago, but you don't
> respond to emails.  
>
> -- Paul Wilson, UW Madison, via Karl Broman
-----------------------------------

## Reproducible research

Reproducibility is a key principle of the scientific method.  To be
credible, an experiment, when performed again by the same
investigator, or a different investigator should give roughly the same
result.  Most bench scientists keep a "lab notebook" to document their
protocols and workflow to facilitate reproducibility.

There is a growing movement bringing the same discipline to 
statistical analysis of data.  The idea is that the output of software
(reports, results, tables, figures) should be reproducible, i.e. other
researchers should be able to reproduce data analysis results.

Recently Nicolas Rougier [issued a challenge](https://rescience.github.io/ten-years/) 
to scientists to reproduce their 10-years or older code.  It highlighted,
in a public way, the importance of [being organized while performing computational
work](https://www.nature.com/articles/d41586-020-02462-7).

In this note, we will outline ideas to organize and document your data
analysis so that you (and others) can reproduce your data analysis.

The points to discuss are:

- Project organization
- Coding style
- Text files
- Command line tools
- Markdown-based report formats

## Project organization

It is helpful to think how you are going to organize your project
files before you start.  There is no one, best way, but here are some
recommendations.

- Make separate directories (folders) for projects
- Make primary data files unwritable
- Separate data from analysis
- Annotate directories using a README file

Here are a few examples.  Directories are indicated by a
trailing `/` and files under it are indented.

------------------------------------
    JaneDoeHW6/
        simpledata.csv
        hw6.Rmd
        hw6.html
        hw6.ipynb
------------------------------------

This is how most homeworks may look like.  The directory should 
have a name that identifies whose homework it is (Jane Doe) and which 
one (HW6).  Inside, there is one simple data file.  You may do the
homework in RMarkdown (`Rmd`) format and generate a report in HTML format.
Alternatively, someone might do the homework in a Jupyter notebook
(`ipynb`).

------------------------------------       
    deadlyInfection/
        data/
            README
            malaria.csv
        analysis/
            analysis.Rmd
            analysis.html
        docs/
            pathBreaking09.pdf
            somewhatRelevant13.pdf
            dataCollection.pdf
------------------------------------       

The `deadlyInfection` directory has a structure I use most often for
my data analysis projects.  It has the data in a
single CSV file.  A `README` file is placed in the directory to 
indicate where the data come from, and how they are coded.
The `analysis`
directory has a single `Rmd` file which is compiled into an HTML format report.  It is assumed to have copious
comments so that it is understandable in the future.  The `docs`
directory has papers relevant for the analysis.

------------------------------------       

    chronicMorbidity/
        primaryData/
            README
            messy.xls
        processedData/
            clean.csv
        analysis/
            README
            Makefile
            python/
                cleaningSteps.py
            R/
                survivalAnalysis.Rmd
                survivalAnalysis.md
                survivalAnalysis.html
        docs/
            relevant.docx
            fromPI.doc
            NEJMPaper.pdf
            
------------------------------------


The `chronicMorbidity` directory has a more elaborate structure where the
primary data has been separated from the processed data.  A python
script is used to clean the primary data, and is placed in the
`python` directory.  R is used to analyse the clean data.  The source
code is in `survivalAnalysis.Rmd` file which is processed by `R/knitr`
(more on this later) into the Markdown file `survivalAnalysis.md`
file, which is then convered into a HTML report file.  All analysis is
done via a `Makefile` in the `analysis` directory.  The `Makefile`
contains a series of instructions to create an analysis pipeline from
primary data to report.

## Coding style: Write good code

This is easier said than done, and beyond the scope of this note, but
it is worth making a few remarks.

- _Comment your code:_ The single most important rule for writing good,
  stable, reproducible, code is to document it as you write.  Err on
  the side of being more elaborate in your comments; it will help you
  understand your code (six months) later, and it will help others use
  and resuse your code.
- _Use relative path names for files:_ This will make your code portable
  across machines and operating systems.  It helps make your code
  reproducible. Avoid using `setwd` in your scripts, and never
  use it with absolute paths.
  For example `setwd("C:/janedoe/bioe806/hw6")` works on your machine,
  but will fail on everyone else's machine. For the homework example
  above, you can simply use `read.csv("simpledata.csv")` and that will
  work everywhere.  It is an example of a relative path. For the chronic
  morbidity example, `survivalAnalysis.Rmd` may have a line like
  `read.csv("../../processedData/clean.csv")`.
- _Make your code look nice:_ Indent code appropriately; insert white
  space and line breaks as needed to make it pleasing to the eye.
- _Use variable names that make sense:_ Try to make variable and
  function names understandable, even if they are long
  (eg. `bonferroni` for a function calculating the Bonferroni
  correction, instead of `bc').  This will help you understand the
  code and avoid making mistakes.
- _Use a
  [convention](https://stackoverflow.com/questions/1944910/what-is-your-preferred-style-for-naming-variables-in-r)
  for long variable names:_ There are three prevalent styles (I
  prefer camelCase):  
  - Period separators: eg. `household.income`
  - Underscore separators: eg. `household_income`
  - Mixed capitalization (camelCase): eg `householdIncome`  
- _Use a consistent naming scheme for variables:_ For example, if you
  are creating versions of a income dataset in sequence, you may name
  the versions in sequence `income1`, `income2`, `income3`, etc.  This
  is simple, but may not tell you how they are different.  So, if the
  first one is for all of the US, the second one a subset for
  Tennessee, and the third one for Shelby County, you may name them:
  `incomeUS`, `incomeTN`', and `incomeShelbyCounty`.
- If you do the same thing two or more times, make a function to do
  that work.
  This will isolate errors, and reduce errors due to
  cut/paste/find/replace mistakes.
- Keep important global definitions such as file/directory names, sample size,
  at the top of a file, so that they are easily visible.
 
## Plain text files: Simple and powerful

Complex file formats (DOC, PDF, HTML, XLS) dominate most of our
electronic lives, but for data analysis, the old-fashioned, plain text
files (TXT, CSV) are best.  Why?

Plain text files are portable across operating systems, and over time.
A text file will be readable in Linux, Windows, and OS X, or your
phone without problems.  You can read them two decades later, but
there is no guaratee that your MS Word file will render correctly on
Windows if it iwas created on a Mac.  We don't know if it will render
correctly two decades from today.

Plain text files take up much less space on your computer; they can
also be read and processed by a wide variety of programs including
your favorite office software.

For data files, a simple CSV (comma separated values) file can record
data as efficiently as a spreadsheet.  It can be read by any
statistical analysis program, and has fewer limitation on the number 
of rows or columns.

For making notes, a plain text file can record information efficiently
with little overhead.  Any word processing program can read it.  To
convey structure in documents and reports beyond unadorned text we
recommend using a simple markup language.  In this note, we will use
[Markdown](https://daringfireball.net/projects/markdown/) that has
been adopted widely by the online community.  Depending on your
preferences, you may also consider
[Asciidoc](http://www.methods.co.nz/asciidoc/), or
[LaTeX](http://www.latex-project.org/).

Finally, tracking changes is also easier with text files.  This may
sound counter-intutive to those using word processors, and will be
clearer to those with experience with version control systems (such as
Git, Mercurial, or Subversion).

After the _content_ of your report is stable, you may convert it to
other formats to share it with others, or for making it look better.
I use [Pandoc](http://johnmacfarlane.net/pandoc/) to convert files to
HTML, PDF, eBooks or word processor formatted files.  The 
[Quarto](https://quarto.org/) tool can also be used to create documents
in a variety of formats.

## Command line tools: A recipe you can follow

Many people find the command line to be unfriendly, and antiquated,
but for data analysis, the command line is your friend.

The main reason is that the command line provides explicit,
reproducible instructions.  You can track what you did, and others can
retrace what you did.  You can also automate repetitive tasks, and
easily make small modifications, without navigating a maze of menus
where a single mistake can cascade down your entire pipeline.

So, whenever possible, use a command line utility for data pre-processing
or data analysis.
<!-- #endregion -->

## Markdown-based report generators

[Markdown](https://daringfireball.net/projects/markdown/) is a text
markup language deigned to be read as plain text, but with indications
regarding its structure.  It is easiest to learn by example.  See the
examples from the [class lectures](https://senresearch.github.io/bioe805/notebooks.html) 
that are in both IPYNB and RMD format.

### Documenting code using R/knitr

[Knitr](http://yihui.name/knitr/) is a R package for report
generation.  You interleave text with R code in a single document.
The R code chunks are marked by a special syntax (that depends on the
format for the text).  Then you can produce a HTML report by using the
function `knit2html`.  Report generation in other formats is also
possible; see the website for details.  In the example repository we
interweave Markdown and R in a `.Rmd` file.

### Using Jupyter notebooks

Jupyter Notebook lets the user create and share live code and
visualizations along with narrative text in the same window. Jupyter
Notebook can be exported into a different format, for instance as
Markdown, PDF, and HTML. However, only the Jupyter Notebook native
format `.ipynb` enable live code.
