Regular boundary element method is employed for the variational formulation of Helmholtz equation that governs the waveguiding problems. Like in the Charge simulation method, in this method, the source points associated with the fundamental solutions are allocated outside the domain so that the singular integrals which occur in the standard boundary element procedure can be avoided. First, the formulation is developed for the two-dimensional scalar Helmholtz problem solving for the axial components of either electric or magnetic fields. The application of the formulation is shown for simple hollow rectangular waveguide and dielectric-slab-loaded rectangular waveguide. Then the formulation is extended for the analysis of dielectric waveguides of open type incorporating axial components of both electric and magnetic fields, for the solution of the propagating modes which are generally of hybrid types. To show the validity and quality of the formulation, it is applied to a circular step-index optical waveguide and a dielectric rectangular waveguide. Very close agreements have been found when the solutions are compared with the ones obtained by different methods. One distinct merit of the extended formulation is that it has been fixed to suppress the spurious solutions which are encountered while solved by the conventional boundary element method.