When a trained cook looks at a chicken breast and says it is done, they are not guessing. They are reading a set of visual signals, built up over years of experience, that map to a reliable internal state. The opacity of the flesh, the golden colour at the surface, the way the skin has contracted, the steam coming off it, the absence of the translucent pink it had before heat was applied. These are sensor readings. They are just not formalised as sensor readings.
One of the things that took us longest in developing Cook-e's classification system was understanding exactly what those signals are and why they correlate with doneness. Not in a vague "colour changes when food cooks" sense, but in a specific physical sense: what is the underlying phenomenon, how does it manifest in the image, and how reliably does it map to the food state we care about?
What happens to food when it cooks
The visual changes in food during cooking are driven by a small number of physical processes. Understanding them is necessary for understanding what the vision system is actually measuring.
For protein-based foods, the dominant visual change is protein denaturation. When proteins in muscle tissue are exposed to heat above approximately 40 degrees celsius, they begin to unfold from their native structure and re-aggregate. This process changes the optical properties of the tissue: raw muscle is partially translucent because the native protein structure scatters light relatively little. Denatured protein is opaque because the re-aggregated protein network scatters light strongly. This transition from translucent to opaque is what a cook sees when they say the flesh "turns white".
The colour change at the surface is a distinct process: Maillard reaction and, in the case of fat-basted surfaces, oxidation. Maillard reaction occurs between amino acids and reducing sugars at temperatures above approximately 140 degrees celsius and produces the golden-brown colour compounds that are visually associated with proper cooking. The reaction is sensitive to surface moisture (too much moisture suppresses it), to surface temperature, and to the sugar content of the food. In a hot-holding cabinet at 70 to 75 degrees celsius, the surface temperature of most items will not reach the Maillard threshold during the hold period; the Maillard colouration that a cook uses as a visual cue develops during the initial cooking step before the item goes into the holding cabinet. The cabinet's vision system is therefore not looking for Maillard browning during holding; it is looking at whether the pre-existing colouration from the cooking step is consistent with the expected profile for a ready item of that food type.
Colour channels: what the camera reads
A standard RGB image sensor captures light intensity in three broad spectral bands: red (approximately 600-700 nm), green (approximately 500-600 nm), and blue (approximately 400-500 nm). The ratio of these channels carries information about the food's surface colour.
For a raw chicken breast, the dominant reflected spectrum is in the red-green range with moderate blue: the pink-white colour of raw poultry. For a cooked chicken breast, the red channel intensity increases relative to the blue channel as the surface transitions to pale gold: red/blue ratio increases significantly. For an overcooked chicken breast, the red channel value continues to rise as surface browning progresses, and texture information becomes critical to distinguish "cooked" from "overcooked".
The challenge with colour analysis is that these ratios are only meaningful under controlled illumination. If the light source spectrum changes between frames, the measured ratio changes even if the food state has not changed. This is why the LED illumination specification described in our earlier hardware post is a first-order constraint, not an afterthought. The colour signal is precise, but its interpretation requires knowing the illumination spectrum.
The model uses per-food-type colour distribution targets rather than a single universal colour threshold. What constitutes a "ready" red/green/blue ratio for a chicken thigh is different from the ratio for a beef patty is different from the ratio for an egg dish. The per-food-type calibration accounts for this.
Texture gradients: the spatial signal
Texture in a camera image is the spatial variation of intensity values across a region. A smooth surface has low spatial variation; a rough or irregular surface has high spatial variation.
Food surfaces change in texture as they cook. Raw protein surfaces are relatively uniform in texture. As protein denatures and the surface dries slightly under heat, the texture becomes more varied: the surface of a cooked sausage has visible irregular features that a raw sausage does not. The model processes the spatial frequency content of a region of interest around each food item on the tray: high spatial frequency content indicates developed surface texture, low spatial frequency indicates an insufficiently cooked or raw surface.
Texture analysis is most useful in two situations. First, when colour information is ambiguous: a food item with sauce coverage may show little colour variation between raw and cooked states because the sauce masks the underlying protein colour. In these cases, texture from the sauce surface itself changes as the food heats (sauces bubble and develop surface features), and this texture change carries doneness information. Second, for distinguishing over-held from properly held: texture continues to degrade as food is held past its optimal window. Surface dehydration creates distinctive texture patterns that differ from a properly held item.
Steam scatter and moisture expression
Steam is harder to measure than colour or texture, but it is a real signal. Hot food under strong illumination scatters light differently from cold food because water vapour above the surface scatters the illumination beam. This is measurable as a local reduction in image contrast around a hot item compared to the same item cold.
The steam signal is most useful as a binary indicator: steam present means the food is above a threshold temperature. It does not carry precise doneness information because steam intensity varies with the food's moisture content, its surface area, and the air circulation in the cabinet. We weight it as a corroborating input in the classification pipeline: if colour and texture indicate a borderline ready state, steam presence increases confidence that the food is genuinely hot and not a cold item that happens to look ready on colour and texture alone (a scenario that can arise if cold food is loaded in a tray adjacent to hot food and the images are processed in rapid succession).
The combination: multi-signal classification
The three signals, colour, texture, and steam, are combined in the model through a feature fusion layer. The model does not make independent decisions from each signal and then vote; it processes all three as a joint feature vector and outputs a probability distribution over the three states: not ready, in hold window, past hold window.
The weighting of the signals is food-type dependent and was set through training. For food types with clear colour change on cooking (most proteins), colour is the highest-weight signal. For food types with strong sauce coverage (casserole preparations, for example), texture is weighted more heavily. For all food types, steam is a supporting corroboration rather than a primary signal.
We want to be careful about one claim: this system is not equivalent to a core temperature probe. A probe inserted into the thickest part of a piece of meat gives a direct reading of the actual core temperature, which is the primary food safety parameter. The vision system reads the surface state, which correlates with the core state but does not directly measure it. The correlation is reliable for common commercial food formats prepared in standard ways. It is less reliable for unusually thick cuts, for items with protective barriers that insulate the core from the surface, or for preparations that are designed to have a discrepancy between surface appearance and core temperature (some pastry preparations, for example).
We do not market Cook-e as a replacement for a core temperature probe as the primary food safety instrument. We market it as a doneness classification system that triggers the hold window timer, alongside a continuous temperature log of the holding environment. These are complementary tools. The classification system is what enables unattended hold management. The temperature log is what satisfies food safety record-keeping requirements. Neither alone is sufficient; together they produce a more complete picture than time-only holding systems currently provide.