Q:

Smoothing binary mask with Hamming/Hanning filter

I have used MATLAB's built-in "imfreehand" to define a binary mask. Every pixel inside the boundary is set to one and otherwise to zero:
h = imfreehand(handles.fig);
wait(h);
mask = h.creatMask();

Then, I want to dilate the mask with 4 pixels, and finally add a piece of code that makes edges smoother, i.e. the edges of the mask smoothly go from 1 to 0; something like Hamming or Hanning window. How can I do that?

A:

so if I understand your question correctly, you have a binary mask, output from h.createMask();, and you want to expand the boundary and then, essentially, 'feather' the mask boundary.
One way I would accomplish this is, more or less like you suggest, to use imdilate() to dilate the mask by a disk with some appropriate radius (say 4). Though I'm not sure it produce exactly 4 pixels dilation. Then I would perform a conv2() with a small kernel. Which could be constructed from a Hann window as you mention. In code:
First, let's produce the mask you are asking about:

% test image
I = imread('cameraman.tif');  % Read a sample grayscale image

% From the OP's Question (suggested edit anyway)
fig = figure;
hAx = axes(fig);
hImg = imshow(I, 'Parent', hAx);
h = imfreehand(hAx);
wait(h);
mask = h.createMask();

Then, we can dilate the mask with imdilate() by creating a larger disk with strel()

%dilate radius of mask by 4
maskExpanded = imdilate(mask, strel('disk', 4));

Finally, we can make a 2D kernel from the hann() window and convolve our dilated mask. The number of points you use for the hann() window controls the amount of feathering. That is, you can increase the feathering by using 20 instead of 10.
%hann kernal
hannWindow = hann(10); %10px
% Convert window into 2 dimensions by outer product.
hannKernel = hannWindow * hannWindow';
% Make the kernel sum to 1
hannKernel = hannKernel ./ sum(hannKernel(:));

% Now Apply Smoothing to the enlarged mask
maskSmoothed = conv2(maskExpanded,hannKernel,'same');

For the sake of completeness, you can view the effects with the following:
%View
f2 = figure; 
f2.Position(3) = 2*f2.Position(3); %twice as wide
f2.Position(1) = f2.Position(1) - f2.Position(3)/2;
f2.Position(4) = 2.1*f2.Position(4); %twice as wide
f2.Position(2) = f2.Position(2) - f2.Position(4)/2;

%axes
ax1 = subplot(2,2,1);
ax2 = subplot(2,2,2);
ax3 = subplot(2,2,3);
ax4 = subplot(2,2,4);
%plots
surf(hannKernel, 'parent', ax1);
imshow(maskExpanded,'parent', ax2);
imshow(maskSmoothed,'parent', ax3);
imshow(maskSmoothed-mask,'parent', ax4);
%titles
title(ax1,'Hann kernel 3d representation');
title(ax2,'Expanded Mask');
title(ax3,'Expanded Mask with convolution');
title(ax4,{'Effect of expansion & convolution';'(difference from original)'});

For me, this creates the following figures:

I would guess there are many ways to do this, but I hope my solution helps.

