This answer demonstrates how to generate a mask by box (range) or ellipsoid (distance) matching in RGB given a target color tuple.
I'm assuming the goal is to do something similar to grayconnected(), so the given example can be further expanded to include sampling from the image and selection of the region local to the sample point.
% inputs
inpict = imread('rainbow.png'); % uint8
tol = [0.2 0.3 0.3]; % pick some tolerance per axis
picklocation = [20 100]; % where to sample the color [x y]
sampleradius = 2.5; % for sample averaging (set to 0 to disable)
% show the image and mark the location
imshow(inpict); hold on
plot(picklocation(1),picklocation(2),'x')
% prepare inputs
inpict = im2double(inpict); % put the image in a consistent scale
tol = permute(tol,[1 3 2]); % reorient tuple along page axis
% sample the image
fk = fspecial('disk',max(sampleradius,0.5));
r = ceil(sampleradius-0.5); % calculated window radius
xw = picklocation(1) + r + (-r:r); % (sample offset) + (padding offset) + (window indices)
yw = picklocation(2) + r + (-r:r);
padded = padarray(inpict,[r r],'symmetric','both');
pickedcolor = sum(fk.*padded(yw,xw,:),1:2); % sample average
% these are global color selections
% perform box (range) selection in RGB
boxmask = all(abs(inpict-pickedcolor) <= tol,3);
imshow(boxmask)
% perform ellipsoid (euclidean distance) selection in RGB
distmask = sum(((inpict-pickedcolor)./tol).^2,3) <= 1;
imshow(distmask)
% restrict the selection to the local blob (magic wand behavior)
conn = 4; % connectivity
boxmasklocal = bwselect(boxmask,picklocation(1),picklocation(2),conn);
imshow(boxmasklocal)
distmasklocal = bwselect(distmask,picklocation(1),picklocation(2),conn);
imshow(distmasklocal)
