Personal blog of Miguel Araujo

Django PositiveNormalizedDecimalField

Some weeks ago I was working on a Django project that was using some FloatFields where they should not be used, the database backend was Postgres 8.4. If you are not aware of it, FloatFields are turned into double precission types, that means 64-bit floating-point numbers. Those fields were storing values that didn’t need such precission. For example I don’t think you need them to store items’ weights, do you?

I migrated those fields to DecimalFields using an average (max_digits=7, decimal_places = 2), which means a total of 7 digits, with 2 decimals. Of course I had to migrate the database, doing some manual alters, as Django-south was not being used. By default DecimalFields render the number with all its decimal digits, which was not what I was looking for. I needed a decimal field that:

  • By default render decimal values normalized using non-scientific notation. See Decimal("10.400").normalize()
  • Only allowed positive values

I could have used floatformat built-in filter, but that would have supposed to change hundreds of different templates. Even though grep and sed are a big allies, the process would be quite error prone. I simply needed a centralized solution. Obviously It was time to create a custom model field, I decided to call it PositiveNormalizedDecimalField. I started reading Writing custom model fields, which gave me a good idea of how things work.

I decided to start subclassing DecimalField and rendering values normalized. I started overwritting to_python method. From the docs I understood to_python was in charge of turning the value stored in the database into a python object. But things were not working. This is the code I had at that time

class PositiveNormalizedDecimalField(models.DecimalField):
    def to_python(self, value):
        if value is None:
            return value
        try:
            # we want to return a normalized non-scientific notation decimal
            return decimal.Decimal('{0:f}'.format(decimal.Decimal(str(value)).normalize()))

However, to_python wasn’t working as django docs prayed. So I thought that maybe Django had a bug, honestly I was pretty much sure I was doing something wrong. So I got into Django’s code and put some breakpoints using pdb, don’t do this in your production server. Then I found out that to_python was not even being called, damn! So I filled a ticket for documentation improvement quite angry after loosing some hours on this.

Thanks to Gabriel Hurley I found out that I needed to use the metaclass SubFieldBase to have to_python being called. That finally made the trick and I got it working. Here is the final code:

class PositiveNormalizedDecimalField(models.DecimalField):
    __metaclass__ = models.SubfieldBase

    default_error_messages = {
        'positive': _('Introduce a positive number.'),
    }

    def to_python(self, value):
        if value is None:
            return value
        try:
            # we want to return a normalized non-scientific notation decimal
            number = decimal.Decimal('{0:f}'.format(decimal.Decimal(str(value)).normalize()))

            # This ensures no negative number can get to the model (model validation)
            if number < decimal.Decimal(0):
                raise exceptions.ValidationError(self.error_messages['positive'])

            return number

        except decimal.InvalidOperation:
            raise exceptions.ValidationError(self.error_messages['invalid'])

After talking to some developers in the django IRC channel I realized adding a field like this into the core was not going to make it. So instead I’m publishing it here and I might be releasing a Django-app in the future with some more model fields.

Some Built-in fields such as PositiveIntegerField have a database CHECK constraint to allow only positive numbers, that constraint is located at django/db/backends/*.py This could be another option to check positive values, however modyfing Django’s code without using official accepted patches is not a recommended option.

This is also a good example in any case you need to redefine default rendering behavior of a model field. Bare in mind, that Django use of to_python is a little bit inconsistent. So when doing aggregate queries to get minimums or maximums, you will need to filter output manually.

blog comments powered by Disqus