Ask Me
Data Preparation
A common task in informatics and other data research workflows is the process of data preparation. Data preparation can include, for example, transforming JSON/XML data from a scholarly API into tabular form, extracting parts of a text file, or replacing characters.
There are numerous workflows for preparing data. Most basic data preparation tasks can be performed using a scripting programming language or even a command line utility such as those within the GNU Core Utilities.
This guide highlights resources for data preparation using Python, the command line, Matlab, and Mathematica.
Python Data Preparation Resources
-
Software Carpentry: Plotting and Programming in PythonThe software carpentry lesson is a good place to start if you have not used Python before. It includes a few data conversion and data loading tutorials.
-
Python Data Science Handbook by Jake VanderPlasIntroduction to data science with Python. Includes a chapter on data manipulation with the Python Pandas library.
-
Python Text Processing ServicesPython documentation for string processing, formatting, and regular expressions.
-
Pandas LibraryPython data analysis library. Includes many data manipulation functions including XML/JSON conversion to a dataframe.
Command Line Tools - Data Preparation Resources
-
Effective AWK programmingAWK is a scripting language for text processing and is commonly used for data extraction and filtering.
-
NCBI EDirect: xtract and transmuteThe NCBI EDirect program contains several command line programs including xtract and transmute, which are designed for working with XML and JSON data.
-
GNU CoreutilsGNU coreutils includes many programs for working with text such as split, sort, cut, tr, and uniq.
Matlab Data Preparation Resources
-
Matlab by Stormy AttawayIntroductory Matlab book. Contains chapters on string manipulation and data structures.
-
Matlab Documentation for Characters and StringsMatlab has many string manipulation functions for matching, replacing, and extracting text.
-
Matlab Documentation: Structured Data and XML DocumentsMatlab documentation for reading XML.
-
Matlab Documentation: Data Import and ExportMatlab documentation for various import and export options including functions for working with spreadsheets and JSON data.
Mathematica Data Preparation Resources
-
Hands-on Start to Wolfram MathematicaIntroductory Mathematica book, includes a chapter on data filtering and manipulation.
-
An Elementary Introduction to the Wolfram LanguageIntroductory book, has several chapters on working with strings, patterns, and data.
-
Wolfram Documentation: Importing and ExportingMathematica documentation, includes many functions for working with text, JSON, and XML.