Ask Me
109 Rodgers Library
205-348-5806
Data Preparation
A common task in informatics and other data research workflows is the process of data preparation. Data preparation can include, for example, transforming JSON/XML data from a scholarly API into tabular form, extracting parts of a text file, or replacing characters.
There are numerous workflows for preparing data. Most basic data preparation tasks can be performed using a scripting programming language or even a command line utility such as those within the GNU Core Utilities.
This guide highlights resources for data preparation using Python, Unix Shell, Matlab, and Mathematica.
Tutorials for Data Preparation
UA Libraries is actively developing tutorials on how to perform basic data preparation tasks using Python, Unix Shell, Matlab, and Mathematica. Our tutorials can be found on GitHub:
Python Data Preparation Resources
- Software Carpentry: Plotting and Programming in PythonThe software carpentry lesson is a good place to start if you have not used Python before. It includes a few data conversion and data loading tutorials.
- Python Data Science Handbook by Jake VanderPlasExcellent introduction to data science with Python. Includes a chapter on data manipulation with the Python Pandas library.
- Python Text Processing ServicesPython documentation for string processing, formatting, and regular expressions.
- Pandas LibraryPython data analysis library. Includes many data manipulation functions including XML/JSON conversion to a dataframe.
Unix Shell Data Preparation Resources
- Effective AWK programmingAWK is a scripting language for text processing and is commonly used for data extraction and filtering.
- NCBI EDirect: xtract and transmuteThe NCBI EDirect program contains several command line programs including xtract and transmute, which are designed for working with XML and JSON data.
- GNU CoreutilsGNU coreutils includes many programs for working with text such as split, sort, cut, tr, and uniq.
Matlab Data Preparation Resources
- Matlab by Stormy AttawayPopular introductory Matlab book. Contains chapters on string manipulation and data structures.
- Matlab Documentation for Characters and StringsMatlab has many string manipulation functions for matching, replacing, and extracting text.
- Matlab Documentation: Structured Data and XML DocumentsMatlab documentation for reading XML.
- Matlab Documentation: Data Import and ExportMatlab documentation for various import and export options including functions for working with spreadsheets and JSON data.
Mathematica Data Preparation Resources
- Hands-on Start to Wolfram MathematicaIntroductory Mathematica book, includes a chapter on data filtering and manipulation.
- An Elementary Introduction to the Wolfram LanguageIntroductory book, has several chapters on working with strings, patterns, and data.
- Wolfram Documentation: Importing and ExportingMathematica documentation, includes many functions for working with text, JSON, and XML.