Clean a CSV of mixed measurements

Convert a measurement column to one chosen unit, preserve original values, and keep ambiguous rows for review.

PythonLast tested Get the complete example ↓
On this page

Run the example

Download the complete source below. In a macOS or Linux terminal, install the SDK, set your secret API key, then run the file.

Install
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install parseapi
API key
export PARSEAPI_KEY='YOUR_SECRET_API_KEY'
Run
python3 clean-measurement-csv.python.py
Use this example

Choose the unit your file needs

Supplier files can mix inches, centimeters, and feet in one height column. This script adds a consistent decimal amount and unit beside each original measurement. It never overwrites a source cell or invents a missing unit.

Save a UTF-8 CSV with a measure column:

csv
id,measure
001,5 ft 11 in
002,180 cm
003,1.75 m
004,180
005,

Install the Python SDK and set PARSEAPI_KEY using the setup above. Choose centimeters for this file:

Terminal
python3 clean-measurement-csv.python.py heights.csv --to cm --output heights-clean.csv

The output is a new file. Omit the input filename to use the included sample.

Convert the source values

Python
answer = parse.measure(value, to=to, locale=locale, system=system)
result = parsed(answer, value)

The target applies to the whole column. Add --locale de-DE only when that is the source's number format, and --system us or --system imperial only when the source's unit meaning is known. The script reuses a result for repeated input under that same context.

Keep review rows in the file

Python
def parsed(answer, original):
    if (not isinstance(answer, dict) or answer.get("measure") != original
            or type(answer.get("valid")) is not bool):
        return outcome("error", "unexpected_response")
    if not answer["valid"]:
        return outcome("review", answer.get("reason") or "unresolved_measure")
    amount, unit, kind = answer.get("amount"), answer.get("unit"), answer.get("type")
    if (not isinstance(amount, str) or not re.fullmatch(r"-?[0-9]+(?:\.[0-9]+)?", amount)
            or not isinstance(unit, str) or not unit or not isinstance(kind, str) or not kind):
        return outcome("error", "unexpected_response")
    return outcome("converted", amount=amount, unit=unit, kind=kind)

The example adds parse_amount, parse_unit, parse_type, parse_status, and parse_reason. Amounts stay decimal strings. 180 receives missing_unit rather than an assumed centimeter unit; a blank input remains missing_measure. Request errors are kept separate from an unresolved measurement. An incompatible target is an error, not a successful blank conversion.

The script validates distinct headers, row widths, output-column collisions, a 200-row limit, and a 64 KiB input limit before making requests. It preserves quoted fields and leading zeros. Requests have a ten-second timeout, no automatic retries, and stop after authentication or rate errors. Exit 0 means all rows converted, 2 means the output includes review rows, and 1 means the file could not be processed.

For larger files and a column-mapping preview, use Bulk. For an input people fill out, build a flexible height field.

Complete example

The complete Python source. The script reads your API key from PARSEAPI_KEY.

Download Python
View and copy the complete source
clean-measurement-csv.python.py
import argparse
import csv
import io
import re
import sys
from pathlib import Path

from parseapi import ParseAPI, ParseAPIError

COLUMNS = ["parse_amount", "parse_unit", "parse_type", "parse_status", "parse_reason"]
SAMPLE = "id,measure\n001,5 ft 11 in\n002,180 cm\n003,1.75 m\n004,180\n005,\n"


def read_rows(stream):
    reader = csv.reader(stream, strict=True)
    fields = next(reader, None)
    if (not fields or "measure" not in fields or any(not key.strip() for key in fields)
            or len(set(fields)) != len(fields) or set(fields) & set(COLUMNS)):
        raise ValueError("Use distinct nonempty headers including measure, without existing parse output columns.")
    rows = []
    for cells in reader:
        if len(cells) != len(fields):
            raise ValueError("Every row must have the same number of cells as its header.")
        rows.append(dict(zip(fields, cells)))
        if len(rows) > 200:
            raise ValueError("Use 1 to 200 rows for this example.")
    if not rows:
        raise ValueError("The CSV needs at least one row.")
    return fields, rows


def outcome(status, reason="", amount="", unit="", kind=""):
    return dict(zip(COLUMNS, [amount, unit, kind, status, reason]))


# example:result
def parsed(answer, original):
    if (not isinstance(answer, dict) or answer.get("measure") != original
            or type(answer.get("valid")) is not bool):
        return outcome("error", "unexpected_response")
    if not answer["valid"]:
        return outcome("review", answer.get("reason") or "unresolved_measure")
    amount, unit, kind = answer.get("amount"), answer.get("unit"), answer.get("type")
    if (not isinstance(amount, str) or not re.fullmatch(r"-?[0-9]+(?:\.[0-9]+)?", amount)
            or not isinstance(unit, str) or not unit or not isinstance(kind, str) or not kind):
        return outcome("error", "unexpected_response")
    return outcome("converted", amount=amount, unit=unit, kind=kind)
# /example:result


def enrich(rows, parse, to=None, locale=None, system=None):
    results, cache, stopped = [], {}, False
    for row in rows:
        value = row["measure"].strip()
        identity = (value, to, locale, system)
        if not value:
            result = outcome("review", "missing_measure")
        elif len(value) > 256:
            result = outcome("review", "measurement_too_long")
        elif identity in cache:
            result = cache[identity]
        elif stopped:
            result = outcome("error", "not_attempted")
        else:
            try:
                # example:request
                answer = parse.measure(value, to=to, locale=locale, system=system)
                result = parsed(answer, value)
                # /example:request
            except ParseAPIError as error:
                result = outcome("error", error.code)
                stopped = error.status in (401, 403, 429)
            except Exception:
                result = outcome("error", "request_failed")
            cache[identity] = result
        results.append({**row, **result})
    return results


def main():
    parser = argparse.ArgumentParser(description="Convert a measurement CSV while preserving its original values.")
    parser.add_argument("input", nargs="?", help="UTF-8 CSV filename; omit for the sample")
    parser.add_argument("--output", help="New output filename; omit for standard output")
    parser.add_argument("--to", help="Compatible target unit, such as cm")
    parser.add_argument("--locale", help="Known source number locale, such as de-DE")
    parser.add_argument("--system", choices=("us", "imperial"))
    args = parser.parse_args()
    try:
        if args.output and (Path(args.output).exists() or Path(args.output).is_symlink()):
            raise ValueError("Choose an output filename that does not exist.")
        if args.input:
            path = Path(args.input)
            if path.stat().st_size > 65536:
                raise ValueError("Use a CSV no larger than 64 KiB.")
            with path.open(encoding="utf-8-sig", newline="") as stream:
                fields, rows = read_rows(stream)
        else:
            fields, rows = read_rows(io.StringIO(SAMPLE))
        # Reads the secret PARSEAPI_KEY from the environment. Never put it in a browser.
        with ParseAPI(timeout=10.0, retries=0) as parse:
            results = enrich(rows, parse, args.to, args.locale, args.system)
        if args.output:
            with Path(args.output).open("x", encoding="utf-8", newline="") as stream:
                writer = csv.DictWriter(stream, fieldnames=fields + COLUMNS)
                writer.writeheader()
                writer.writerows(results)
        else:
            writer = csv.DictWriter(sys.stdout, fieldnames=fields + COLUMNS)
            writer.writeheader()
            writer.writerows(results)
    except (ValueError, OSError, csv.Error) as error:
        print(f"Could not complete: {error}", file=sys.stderr)
        return 1
    review = sum(row["parse_status"] != "converted" for row in results)
    print(f"{len(results)} rows written, {review} need review.", file=sys.stderr)
    return 2 if review else 0


if __name__ == "__main__":
    sys.exit(main())

Need a key? Set up this server example. For every field and parameter, see Measure API reference.

Build something else

All tutorials →