Clean a CSV of mixed measurements

Convert mixed measurements in a Python CSV to one chosen unit, preserving decimal strings and unresolved rows.

PythonLast tested Download and run ↓
On this page

Run the example

Download Python

clean-measurement-csv.python.py

Download the file, then run these commands in its folder using a macOS or Linux terminal. You need Python 3.9 or later and a secret API key.

Terminal
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install parseapi
export PARSEAPI_KEY='YOUR_SECRET_API_KEY'
python3 clean-measurement-csv.python.py --to cm

The first run uses the included sample. The CSV instructions below show how to save your own results. Review rows can produce exit code 2 even when the output was written successfully.

Need a key? Set up this example

Choose the unit your file needs

A supplier's height column might mix 5 ft 11 in, 180 cm, and 1.75 m. Use Python to convert those measurements to one unit while keeping the original cells. Values without a unit stay in the file for review.

Save this as heights.csv, using a UTF-8 CSV with a measure column:

csv
id,measure
001,5 ft 11 in
002,180 cm
003,1.75 m
004,180
005,

Choose centimeters as the target for this file:

Terminal
python3 clean-measurement-csv.python.py heights.csv --to cm --output heights-clean.csv

The output is a new file. Omit the input filename to use the included sample.

Convert the source values

Python
answer = parse.measure(value, to=to, locale=locale, system=system)
result = parsed(answer, value)

The target applies to the whole column. Add --locale de-DE only when that is the source's number format, and --system us or --system imperial only when the source's unit meaning is known. The script reuses a result for repeated input under that same context.

Keep review rows in the file

With --to cm, selected output columns look like this:

csv
id,measure,parse_amount,parse_unit,parse_status,parse_reason
001,5 ft 11 in,180.34,cm,converted,
002,180 cm,180,cm,converted,
004,180,,,review,missing_unit
005,,,,review,missing_measure
Python
def parsed(answer, original):
    if (not isinstance(answer, dict) or answer.get("measure") != original
            or type(answer.get("valid")) is not bool):
        return outcome("error", "unexpected_response")
    if not answer["valid"]:
        return outcome("review", answer.get("reason") or "unresolved_measure")
    amount, unit, kind = answer.get("amount"), answer.get("unit"), answer.get("type")
    if (not isinstance(amount, str) or not re.fullmatch(r"-?[0-9]+(?:\.[0-9]+)?", amount)
            or not isinstance(unit, str) or not unit or not isinstance(kind, str) or not kind):
        return outcome("error", "unexpected_response")
    return outcome("converted", amount=amount, unit=unit, kind=kind)

The example adds parse_amount, parse_unit, parse_type, parse_status, and parse_reason. Amounts stay decimal strings. 180 receives missing_unit rather than an assumed centimeter unit; a blank input remains missing_measure. Request errors are kept separate from an unresolved measurement. An incompatible target is an error, not a successful blank conversion.

The script validates distinct headers, row widths, output-column collisions, a 200-row limit, and a 64 KiB input limit before making requests. It preserves quoted fields and leading zeros. Requests have a ten-second timeout, no automatic retries, and stop after authentication or rate errors. Exit 0 means all rows converted, 2 means the output includes review rows, and 1 means the file could not be processed.

For larger files and a column-mapping preview, use Bulk. For an input people fill out, build a flexible height field.

Complete example

The complete Python source. The script reads your API key from PARSEAPI_KEY.

View and copy the complete source
clean-measurement-csv.python.py
import argparse
import csv
import io
import re
import sys
from pathlib import Path

from parseapi import ParseAPI, ParseAPIError

COLUMNS = ["parse_amount", "parse_unit", "parse_type", "parse_status", "parse_reason"]
SAMPLE = "id,measure\n001,5 ft 11 in\n002,180 cm\n003,1.75 m\n004,180\n005,\n"


def read_rows(stream):
    reader = csv.reader(stream, strict=True)
    fields = next(reader, None)
    if (not fields or "measure" not in fields or any(not key.strip() for key in fields)
            or len(set(fields)) != len(fields) or set(fields) & set(COLUMNS)):
        raise ValueError("Use distinct nonempty headers including measure, without existing parse output columns.")
    rows = []
    for cells in reader:
        if len(cells) != len(fields):
            raise ValueError("Every row must have the same number of cells as its header.")
        rows.append(dict(zip(fields, cells)))
        if len(rows) > 200:
            raise ValueError("Use 1 to 200 rows for this example.")
    if not rows:
        raise ValueError("The CSV needs at least one row.")
    return fields, rows


def outcome(status, reason="", amount="", unit="", kind=""):
    return dict(zip(COLUMNS, [amount, unit, kind, status, reason]))


# example:result
def parsed(answer, original):
    if (not isinstance(answer, dict) or answer.get("measure") != original
            or type(answer.get("valid")) is not bool):
        return outcome("error", "unexpected_response")
    if not answer["valid"]:
        return outcome("review", answer.get("reason") or "unresolved_measure")
    amount, unit, kind = answer.get("amount"), answer.get("unit"), answer.get("type")
    if (not isinstance(amount, str) or not re.fullmatch(r"-?[0-9]+(?:\.[0-9]+)?", amount)
            or not isinstance(unit, str) or not unit or not isinstance(kind, str) or not kind):
        return outcome("error", "unexpected_response")
    return outcome("converted", amount=amount, unit=unit, kind=kind)
# /example:result


def enrich(rows, parse, to=None, locale=None, system=None):
    results, cache, stopped = [], {}, False
    for row in rows:
        value = row["measure"].strip()
        identity = (value, to, locale, system)
        if not value:
            result = outcome("review", "missing_measure")
        elif len(value) > 256:
            result = outcome("review", "measurement_too_long")
        elif identity in cache:
            result = cache[identity]
        elif stopped:
            result = outcome("error", "not_attempted")
        else:
            try:
                # example:request
                answer = parse.measure(value, to=to, locale=locale, system=system)
                result = parsed(answer, value)
                # /example:request
            except ParseAPIError as error:
                result = outcome("error", error.code)
                stopped = error.status in (401, 403, 429)
            except Exception:
                result = outcome("error", "request_failed")
            cache[identity] = result
        results.append({**row, **result})
    return results


def main():
    parser = argparse.ArgumentParser(description="Convert a measurement CSV while preserving its original values.")
    parser.add_argument("input", nargs="?", help="UTF-8 CSV filename; omit for the sample")
    parser.add_argument("--output", help="New output filename; omit for standard output")
    parser.add_argument("--to", help="Compatible target unit, such as cm")
    parser.add_argument("--locale", help="Known source number locale, such as de-DE")
    parser.add_argument("--system", choices=("us", "imperial"))
    args = parser.parse_args()
    try:
        if args.output and (Path(args.output).exists() or Path(args.output).is_symlink()):
            raise ValueError("Choose an output filename that does not exist.")
        if args.input:
            path = Path(args.input)
            if path.stat().st_size > 65536:
                raise ValueError("Use a CSV no larger than 64 KiB.")
            with path.open(encoding="utf-8-sig", newline="") as stream:
                fields, rows = read_rows(stream)
        else:
            fields, rows = read_rows(io.StringIO(SAMPLE))
        # Reads the secret PARSEAPI_KEY from the environment. Never put it in a browser.
        with ParseAPI(timeout=10.0, retries=0) as parse:
            results = enrich(rows, parse, args.to, args.locale, args.system)
        if args.output:
            with Path(args.output).open("x", encoding="utf-8", newline="") as stream:
                writer = csv.DictWriter(stream, fieldnames=fields + COLUMNS)
                writer.writeheader()
                writer.writerows(results)
        else:
            writer = csv.DictWriter(sys.stdout, fieldnames=fields + COLUMNS)
            writer.writeheader()
            writer.writerows(results)
    except (ValueError, OSError, csv.Error) as error:
        print(f"Could not complete: {error}", file=sys.stderr)
        return 1
    review = sum(row["parse_status"] != "converted" for row in results)
    print(f"{len(results)} rows written, {review} need review.", file=sys.stderr)
    return 2 if review else 0


if __name__ == "__main__":
    sys.exit(main())

Need a key? Set up this server example. For every field and parameter, see Measure API reference.

Build something else

All tutorials →