For anyone else trying to do this, you can get the field key with:
$field.data('key')
Now you can do something like this:
if ($field.data('key') == 'field_609a8f6d92e06') {
args.palettes = [ "#F00", "#0F0", "#00F" ];
} else if ($field.data('key') == 'field_60a64e4b2f452') {
args.palettes = [ "#FF0", "#0FF", "#F0F" ];
}